Accessible Music

After much contemplation about what to do for my project, I have landed on my final project concept: Accessible Music. Accessible, meaning that it can be useful for people with disabilities (in particular, the deaf community). The idea of this project is to provide a way to enhance the experience of music for people with hearing disabilities.

Project Idea

For my Senior Project, I wanted to delve into this topic of accessible music and make something that can potentially benefit non-music people and deaf people. In regards to non-music people, I find that many people shy away from learning music because it is either too time consuming or they think it is too late to start and would be too difficult. I want to make music easier to experience and easier to learn. This way, everyone can enjoy music without any limitations.

My project will include a musical piece accompanied by a visual representation of each individual note and timbre with colors and tone. The goal for this is for someone to be able to identify the instruments and notes used just by listening to it (as opposed to reading the sheet music or seeing the Logic file). In addition to this, there will be an active interaction element to my project. Through the use of a MIDI keyboard, I will allow the user to play around with the notes and timbres so that they themselves can see the difference in colors with each note and instrument. Ideally, a deaf person will be able to learn the difference between notes and timbres and see how music is represented visually. Through the use of color, shapes, tints, and shades, I will visually represent music in a way that can benefit people with no music experience (or hearing experience).

Inspiration

Originally, my inspiration for this project stemmed from synesthesia, the body’s ability to link one sense to another. Specifically, I have always been intrigued by chromesthesia, the link between sight and sound. For people who have chromesthesia, they see colors and certain shapes every time a musical note or sound are heard. Typically, these people also have perfect pitch, another phenomena that fascinated me. Perfect pitch is the ability to recognize any pitch without a reference pitch or key. Since I have spent my whole life surrounded by music performance and theory, I kept wishing and dreaming that one day I will wake up and be blessed with perfect pitch and chromesthesia. “All I want is to see colors when I hear notes and be able to identify those notes. Is that so much to ask?” Apparently not. Anyone can actually train themselves to have perfect pitch and synesthesia. All it takes is a lot of training and associations between certain colors and notes. After doing a bit of research I realized that chromesthesia is not everything I thought it was.

I watched a YouTube video of this woman who has chromesthesia and how she sees music. In my head, I always envisioned chromesthesia being a wonderful rainbow of colors and shapes that makes life become one giant art piece. In the video, I noticed that instead of rainbows, I saw a few colors projected at different parts of the screen. There were a few main colors (green, yellow, blue), and they would change their position on the screen based on how high or low the note was. Even though I have spent years training my aural skills, I could not identify the notes and their relation to each other. I was partially heartbroken that my lifelong dream was not the reality which stood before me on a YouTube video.

Lost and confused, I wondered what I should do for my senior project. Should I be true to synesthesia and accurately represent what it should look like? Or should I do it my way and show a variety of colors and tones to accurately represent notes? I officially decided that being able to visually represent each note accurately is more beneficial than staying true to synesthesia. Instead, I am considering this project an inspiration from visual music; a way to experience music with more than one sense.

Tl;dr: Synesthesia originally inspired my project. As time went on, I was more inspired to accurately represent notes to benefit people with no music/hearing experience.

Originality and Usage

I can see my project being beneficial outside of the Senior showcase. With a lot of projects I have seen involving music and visuals, it’s more so along the lines of “oh, that looks nice.” Sure, it’s cool to look at, but wouldn’t it be so much more exciting if people can actually benefit from it? It’s one thing to make a project that “looks nice.” It’s a totally different story if the project looks nice AND has a use other than its visual aesthetics. As I’ve discussed before, I would ideally want deaf people to be able to use my project as a way to experience music. Since deafness prevents them from being able to hear music, I want to draw on one of the other senses to enhance their musical experience. Not only does their experience change, but they can also potentially learn music more easily by having the notes visually represented. Of course, this project is not limited to the deaf community. I think that having a visual representation of music can make the learning experience of music that much easier for anybody. I see a potential use of my project for almost everybody across the spectrum. For those that aren’t interested in learning music or already know music, it’s visually and audibly pleasing. For those that want to learn music, it makes the learning experience easier. For those that are deaf, it completely changes their experience with music and can provide a useful way to learn music without hearing it. From those that have no interest or skill in music, to those that spent years learning music, to those that physically cannot hear music, I see a benefit that visual music can have on each of these groups.

In the past, there have been countless examples of visual music throughout history. At concerts, we see colored lights associated with certain songs and moods all the time. Visual music has been done before. But has it been catered to the deaf community? From my understanding and research, there has been no solid evidence towards accessible music for deaf people. Technically my idea for visual music is not original, but my goals for this project are original and have not been implemented before.

Technical Implementation

Although I am still in the beginning phase of this project, I have some ideas of have I can create this project technically. For a prototype, I might use a Makey Makey and assemble a Max patch to match certain pitches and colors. The idea is that each note will have its own color. For example, C can be blue, where C1 will be a navy blue and C8 will be a baby blue. Also, each MIDI timbre will have its own tint or hue to it. For example, a violin may have a brownish hue to it, whereas a flute may have a blueish hue to it. If a violin played a C4, it will look like a royal blue mixed with some brown undertones. For the final project, I may use a MIDI keyboard to make the interactivity more user-friendly. As for the musical piece, ideally I would like to compose a piece showcasing a variety of different instruments and notes. If this proves to be too difficult and time consuming, I can use an existing musical piece not bound by copyright, so long as I have the score. Once I have the piece, I will create a visual representation of the notes and instruments through Aftereffects, with the colors and tones being the same as the ones used in the interactive segment. First and foremost, I will create the interactive element since that is the most important part of this project. After that, I will work on the visual music demo.

After much internal debate and indecisiveness, I have finally landed on a thesis project idea: Cultural Map through the Ages.

Now, this idea was the ultimate winner for me since after peer feedback it was clear how unique the culture map was while also calling to mind that one of the goals for senior thesis is to have a new and innovative idea. As for the why of it all, this project idea stemmed from my own personal interest in history and anthropology. With a main interest in early A.D./B.C. civilizations, I like to learn about myths, folk tales, daily lives, civilization structure, etc. This map will take information from myriad civilizations and display it in a fun, interactive format. While highlighting personal interest and values, I also hope to showcase future goals with this project by presenting my familiarity with technology and showing how my project can reflect what I as a person can accomplish.

One question that’s bound to be asked is “Why should anyone care about this project” during my presentation. Well, it might not be for everyone, I acknowledge that much, however it shows how we can take history, a subject usually reviewed in boring lengthy textbooks and online articles, into a fun, easy to present manner. A phrase I’m quite fond of is “knowledge is power” and through my thesis project I hope to impart onto others the knowledge of ages past. In the current field of historical civilizations and information on them, all I’ve found so far are web pages and the occasional video, however at one point I stumbled upon an interactive map for the United States that had very brief information, however the style in which it was presented left an impact. I plan to take the boring and dreary and making it fun and interactive. Now, as to how this “extends knowledge rather than duplicating” is a bit difficult to answer since the topic at hand involves history, events which have already occurred and cannot be added on to. So rather than extending the content itself, I’m taking the method in which it is conveyed to the general public and making a change on that.

Speaking of how I’ll bring it to the public, let’s talk about the more technical aspects. Nothing is 100% set in stone, however I was planning on having a web page dedicated to this while having an interactive globe through which users can zoom in on certain areas and click for information given about that country’s group of people at the designated time. I am still working out this stage of it as I have only just recently decided to take up this idea. Some prototypes I’m thinking of at the moment involve a sketch for how I want the layout to look and a sketch showing how the interaction will work step by step, among other ideas so I can nail down the step by step procedures and how I’ll have to implement these.

I’ve been booking concerts/shows featuring local bands for a couple of years now. After realizing that it is sometimes difficult to stay connected with the local music scene, my friends and I shared the idea of starting our own label. This label, or for better words-collective, would represent local music artists and help music supporters more easily get connected to the music scene. There are currently very disorganized outlets for this, such as a group on facebook called “NJ DIY”. I have often searched for local concerts using this group, only to find myself confused and wishing there was something easier to understand. This app could bring everything together into a social media outlet to connect musicians with supporters in order to have more of a presence and include other very necessary elements (discussed below).

The target person for my app would be a young music lover or musician. (the age could potentially vary) The app would initially have a welcome page with a title that has something to do with local music (have not figured out a title/logo yet), a small “about” section, and a sign up button. The sign up selection will take you to a basic profile creation page that asks for a specific location- whether you are a music lover or a musician/band. Once the user selects their state, they will be able to view a selection of categories. The categories (in a hamburger type of tool) would include: artists, calendar, forum, live, & feed. The artists section would be an “encyclopedia” of local artists names and information. To be a part of this encyclopedia, bands would have to sign up on the simple portfolio screen. This band profile could potentially list everything from just info on the bands, to their tour dates, to merch. The calendar would list dates of local shows coming up with locations. People could input these when they are aware of shows coming up. These specific events could be laid out similar to a facebook event page with a graphic, info, times, and comment section. The forum would be intended for discussing upcoming shows, musicians hoping to borrow/use equipment, connecting singers with guitar players/bands, connecting volunteers with promoters, etc. The photo/feed and live sections are categories that I’m not completely sure about implementing yet. The photo/feed section could have a similar layout to instagram that displays photos from previous events on the calendar. The live section would be a live broadcast (similar to periscope) of events listed on the calendar, that anyone can post on. All of these categories are interactive and can be inputted by musicians or fans.

I’ve always wanted to work in the music industry while combining my love of digital media and creative work. This app would be a great platform to present my skills, as well as include many different creative platforms that I enjoy. I believe that future employers, as well as local music supporters would love my app idea. I have pitched the idea to coworkers in the industry before and they seem to be excited by the plans and almost confused that it didn’t already exist. The local music community is looking for something like this, just as I was and could use it for their convenience. There are only a few apps similar to this, such as bandsintown, that don’t include nearly as much interaction, such as the forum, and ability to update listings as mine.

To make my project come to life, I’m going to need to do a ton of researching, surveying, and communicating with people who are experienced in app design and development. I don’t have very much experience in this, besides UXD, but am going to spend this and next semester using Lynda and other online services to acquire the knowledge possible to create this thing! For specific prototypes, I will start with drawing up and then wire-framing my categories pages. Stylistically, after selecting your location, a hamburger tool will appear with the categories listed above. My prototype will show what happens when each category is selected and what screen shows up. I will create wireframes and designs using the program: sketch (or whatever I feel works the best). Hopefully next time I can start drawing up some of my ideas. 

After our class discussions I’ve decided to create the VR platform rhythm game that would involve the player moving along a set path and turning where there are changes within the music. Music has been my main focus throughout my years as an IMM major but it seemed like it would be a cop out to just create a music album for my thesis especially when the project has to be interactive. Although this would be my first attempt at creating a game, let alone a VR game, a Rhythm game seemed like the best way to immerse the user in the music while they are also having fun playing the game. My main inspiration for this game was the PS4 game Beat Saber which is a similar rhythm game except the user has to cut through objects moving towards them.

I’m currently in an interactive music class with Dr. Nakra where we are studying how the music in games affect the gameplay. In almost all of these games you see that the music plays a huge part in the user experience. Games like the new Spiderman on PS4 and Assassin’s creed all have unique musical themes that make the user more immersed in the game whether they are at a dangerous point in the game or at a heroic point while others like The last of Us use the lack of music at certain points to make the mood even more eerie. However what these games have in common is that the visual aspect drives the music. This game would be doing the opposite of that.

The user would have to listen to the music and discern the right times to turn to avoid falling off a platform or running into a wall. I think users would enjoy the experience because not only would they be playing the game but they would be listening to music and actually start recognizing where and how the changes happen and might even notice these patterns in other music they listen to. For this demo I’m thinking of creating 1-3 levels, depending on how much I can do within the time allotted to us, with around 1-2 minutes of music per level that I will be making myself and have actually already started composing.

Unlike beat saber where the objects are coming towards you, on each level the user would be moving forward having to turn right or left at the right times and the speed would be determined by the level that they are playing. My idea is to create a hallway that the user would be moving through with an artifact to collect at the end of the level. It would have to be completed in one try and if the user hits the wall, they have to restart from the beginning. Some possible prototyping would be to actually play beat saber and get a feel for the game and it’s environment, animation of the hallway environment and testing the movement through the game. I would also have to get acquainted with the Unity engine by creating a test environment as that is the platform I’m going to be using to create this VR rhythm Game.

My final idea for my thesis is to take the base game of Bethesda’s Fallout 4 and create within it a full-length original quest mod that will bring the player to a new location with a fresh narrative and a unique form of gameplay that’s new to Fallout 4. Inspired by my other favourite post-apocalyptic series, Metro 2033,  this quest will bring the player to a vast underground cavern with strange NPCs, exotic radioactive flora, and deadly new creatures. I’d like to recreate the sense of wonder that Skyrim’s Blackreach had combined with Metro’s sense of constant dread. This theme will be completely new for the base Fallout 4 game and a fresh idea in the modding community. Plus, I’d like to get into some scipting that involves stealth. Base Fallout 4 isn’t much of a stealth game, but the mechanics are there. With some base scripting, I may be able to enhance the stealth that heightens the level design.

 

I was inspired to take up this idea by the years and years of watching this goofy dude, AlChestBreach, play mods for Bethesda games. I’ve even downloaded and played some of them myself. For a while, I’ve wanted to give mod creation a shot myself. Since game design and model creation is the direction I’m taking for my future, modding seems like great practice to me. With my idea for a project, I’ll be able to follow many paths that game development offers and truly see what is a good fit for me. Plus, with NPC creation, I’ll be able to practice my talent in voice acting. Including in this mod a narrative experience will also allow me to flex my talents in creative dialogue writing.

I think that anyone even a little interested in first person shooters/stealth games will have an interest in this mod. Plus, the great thing about mods is that you can upload them on a site like https://www.nexusmods.com/fallout4/ and anyone who owns a copy of Fallout 4 can download and play your creation for free. Hopefully, once on the site this mod will really stick out for how differently it treats the base game. Not only are there typical weapon and NPC additions, but it will completely change how you play. You see, base Fallout 4 is basically a sandbox with a whole lot of shooting and looting and that’s about it. My hopes with this mod are to change the game to be more centered around thinking before shooting.

The majority of this project will involve what is known as the G.E.C.K. This is the game editor released to the public by Bethesda that is used by them to create these games. Without this, creating complex narrative mods would be impossible. Also needed is modeling software like Maya and texturing software like photoshop, both of which have experience in. Some possible prototypes may involve the testing of my stealth ideas, simple NPC test creation, and the insertion of weapon models and animations.