Sheet Music Creator App: Implementation Description
Jim Stys, Erica Gucciardo, Nick Celik, Keith Kraus
I Pledge my honor that I have abided by the Stevens Honor System.
Jim Stys / Erica Gucciardo / Nick Celik / Keith KrausPercentage of effort towards this assignment / 25 % / 25 % / 25 % / 25 %
Section I: Implementation Block Diagrams and Best Design Analysis
Mobile OS Portability
1. Android App & iOS App: These portions have already been assessed in the initial block diagram
2. Proguard:
a. Our Android application, programmed in JAVA, will contain what is called a 'source jar'. This is basically the collection of source files of our program/app. Alot of these source files are created automatically and are in general completely unused. If we translated this code as well during the conversion, the iOS app would become unnecessarily bloated.
b. Here comes into effect the open source tool ProGuard, which will help you shrink, obfuscate, and otherwise mangle Java bytecode. We will give ProGuard a bytecode jar, whereupon it will print out a report listing all of the unused classes and methods in the android app.
c. J2ObjC will then use this report to skip the unnecessary classes and methods during translation, thus reducing bloat and possible translation errors.
3. J2ObjC Translation Steps:
a. Rewriter: “...modifies the initial Java AST model to make it more Objective C friendly. It was the first translation class, and over time most of its functionality was moved to separate translation phases as their complexity increased”
b. Autoboxer: “...converts primitive references into their equivalent numeric wrappers (boxing), and vice versa (unboxing). The converted AST is very similar to how javac modifies its AST, with the same performance.”
c. iOS Converter:
I. Type Converter: “...translates references to those types that are replaced by Foundation framework classes. Array classes are also replaced with native replacements included with the JRE emulation library; these native classes have a "IOS" prefix...”
II. Method Converter: “...updates method references to refer to iOS replacements. It does this using a mapping table, JRE.mappings, which declares the methods to replace and their replacement.”
d. Initialization Normalizaer: “...moves initializer statements into constructors and class initialization methods. This includes expressions used to initialize fields and static variables, as well as static and instance blocks.”
e. Class Converter:
I. Anonymous Class Converter: “...modifies anonymous classes to be inner classes. This involves adding fields for the final variables referenced by that class, which are stored as final fields in the class. If there are any final fields, a constructor is added to the class which has those fields as its parameters. If there is an initialization block (not common), it's statements are added to the new constructor.“
II. Inner Class Converter: “pulls inner classes out of their containing class, and makes them top-level classes in the same compilation unit as their containing class.”
f. Destructor Generator: “...creates memory management-specific destructor methods as needed”
Server Technology – Remote and Local
· Local – Windows Server 2008
o Active Directory
§ Functionality
· Authentication for users to allow for secure uploading and downloading using the application
· Authentication for users to allow for secure storage of converted sheet music and original audio in cloud storage
· Means of digital signature for copyright protection of user submitted audio and sheet music
§ Interfaces
· LDAP for communication from the application to the Active Directory and vice versa
· LDAP for communication from the cloud storage to the Active Directory and vice versa
· Kerberos Authentication used for digital signature of submitted audio and sheet music of users
§ Performance Metrics
· Processing power used by hour, day, week, etc. for the Active Directory
· Number of queries done on the Active Directory per hour, day, week, etc.
· Average speed of query per hour, day, week, etc.
· Time taken to pass authentication through LDAP to application, cloud storage
· Audio Processing Application
o Functionality
§ Converts submitted audio into sheet music
§ Uploads both submitted audio and converted sheet music into cloud storage
§ Digitally signs the files for copyright protection
o Interfaces
§ LDAP for authentication from the Active Directory
§ LDAP for authentication to the Cloud Storage
§ Internet data transfer of audio files from mobile application into this application
§ Internet data transfer of audio files and converted sheet music into the cloud storage
o Performance Metrics
§ Processing Power used for audio to sheet music conversion
§ Average time taken to convert audio to sheet music
§ Processing Power used to digitally sign audio and sheet music files
§ Average time taken to digitally sign files
§ Bandwidth used importing audio files from mobile application
§ Bandwidth used exporting audio files and converted sheet music to cloud storage
Option 1: Cloud Audio Processing Option 2: Local Audio Processing
Relay data back to
Local device
Pitch Detection from Input Audio (mic)
o Functionality
§ Determine fundamental frequencies from potentially polyphonic audio admitted to the system through the peripheral’s microphone
§ Perform a fast fourier transform followed by Cepstrum analysis to produce fundamental frequencies
§ Quickly lookup the translation between frequency and the corresponding musical note(s)
o Interfaces
§ Android/iOS audio peripheral (only the embedded microphone, line-in is hardware dependent)
§ Audio stream that is handled by buffers in the KissFFT library
§ Fundamental frequency array that is processed by the Translation Table
§ GUI interface à Translated musical notes and timing data are sent to the GUI interface layer to be translated into musical notes on a staff
o Performance Metrics
§ Speed of the FFT library used (KissFFT vs. other options)
§ Efficiency when implementing Cepstrum analysis on top of the KissFFT library
§ Sampling rate of the hardware microphone on the peripheral
· GUI Development and Design
o Functionality
§ Get user input and show messages/screens
§ Capture audio using mic on device
§ Send and receive information from server and display appropriate screens
o Interfaces
§ Connection to Server
§ Interaction with User
§ Audio acquisition from Microphone
§ Interface with core code
o Performance Metrics
§ Speed and responsiveness of interface
§ Appealing and aesthetically satisfying
§ Amount of users who complete the process Versus amount of users who exit the application
§ Rating on the App store
Section II: BEST Solution
There are two main options that the group is presenting as an implementation for the sheet music creator app. The first involves using remote server technology by harnessing the Google cloud engine for remote audio processing and storage of user created files. The second is a more simplified local-only model where all processing is done on the local mobile device and all files created by the user are stored locally as well. Both have their advantages and disadvantages but the local-only model is the best solution at least for the beta / early releases of the application. The local-only model does not incur the extra expenses of needing servers for processing and storage. This is a better choice for the launch of this product because it will determine if there is likely to be a future demand for such infrastructure. The app will be flexible enough so that any online functionality and remote server technology can be implemented directly on top of the local-only model, so while using a remote server and the Google cloud engine would be the better implementation, using the local-only model suffices for initially launching the product due to economic constraints.
Section III: Constraints and Responsibilities
LDAP/AD Authentication – User Login and Secure Cloud Storage
· Economics
o A server that is responsible for only an Active Directory does not need to be overly powerful as the Google Cloud Engine will do all of the heavy processing for the audio conversion. For this reason, a moderately powerful server that costs around $1500 - $2000 should be more than sufficient.
· Safety
o The Active Directory would have to be thoroughly secured to prevent users from being able to abuse the server. Furthermore, it would have to be secure to prevent users from hijacking other users’ accounts / subscriptions.
· Sustainability
o Daily maintenance and service will have to be done on the Active Directory to ensure that it’s running smoothly and efficiently. A plan will also need to be formed to be able to migrate from an Active Directory platform to a different platform if the need arises.
Server-Side Audio Processing
· Economics
o Audio processing of this magnitude could involve a very complex algorithm that requires a large amount of processing power to run in the time required. This level of processing power will cause significant expense to the already costly server(s).
o The Google Cloud Engine provides a very scalable model for reasonable prices so that the services can scale from hundreds to millions of users with relative ease.
· Safety
o There needs to be a form of copyright protection to ensure that the application isn’t being abused to steal someone else’s work or gain tabs / sheet music for a song that is legally copyrighted. A system will also need to be in place where users can claim ownership of a piece of music they’ve submitted to prevent other users from attempting to claim it as their own.
· Manufacturability
o The application that does the audio processing would probably take between 6 months and a year to develop, which is a significant amount of time. Once the software is released, patches can be released every 2 to 4 weeks for security and other fixes. It will be of utmost importance to maintain stable software.
· Sustainability
o The application must scale well to a large user base without becoming inefficient or ineffective. Furthermore, the application will have to be able to be ported to newer versions of operating systems or different operating systems with relative ease if possible.
Pitch Detection Algorithms
· Economic
o The use of external libraries depends on their availability to projects such as the one being developed for sheet music generation. The group must consider different options and whether the licenses are accommodating enough and affordable enough (if not free) to be included in the project.
· Security
o Access to different components of the smartphone hardware i.e. the audio recording interface or user storage must be done only at user request so that the user does not regard the application as malicious.
· Professional and Ethical Responsibilities
o Similar to the Economic constraints, all the terms of the license that is being acquired need to be agreed to and adhered to throughout the project.
o The application must inform users that access to microphone or user data will not be used for malicious purposes and initiated only when the user pro
Section IV: SWOT Analysis
Strengths
· Stores and processes all data locally so that no creative property can be stolen or infringed upon between users.
· Cost-effective for an application that has an unknown demand. If cloud servers were needed from launch, it might not be worth the investment.
· First application of its kind to attempt the task of auto sheet music generation
Weaknesses
· Relies on processing power on a mobile device which for the time being is not on the same level as a dedicated server.
· Little interaction between users. No online capabilities means less sharing of creative material which can often be an integral part of music creation
· Relies on FFT algorithms that are not the most accurate and the difference between one pitch and the next can be as small as 1 Hz. Prone to error.
Opportunities
· Cloud integration and online functionality could be implemented on top of the local-only model without disrupting the working application. Expansion would depend on user demand.
· Moving to cloud processing would make the porting from Android to iOS platform simpler because there would be no need for a specific audio processing library on the local device.
· Portability to the iOS platform is still very possible using the research on OS portability (see above).
Threats
· Other applications with more accurate, faster audio processing algorithms
o Important to keep this implementation separate so that it can easily be modified / improved
· Hacking user accounts / data if online functionality is implemented.
Section V: References Appendix
Mobile OS Portability References
http://code.google.com/p/j2objc/wiki/JavaConversions
http://code.google.com/p/j2objc/wiki/DesignNotes
http://code.google.com/p/j2objc/wiki/DeadCodeEliminator
http://code.google.com/p/j2objc/wiki/IosMethodConverter
http://code.google.com/p/j2objc/wiki/InitializationNormalizer
http://code.google.com/p/j2objc/wiki/AnonymousClassConverter
http://code.google.com/p/j2objc/wiki/InnerClassConverter
http://code.google.com/p/j2objc/wiki/DestructorGenerator
Server Technology – Remote and Local References
http://technet.microsoft.com/en-us/library/cc773309(v=WS.10).aspx
http://technet.microsoft.com/en-us/library/cc755809(v=WS.10).aspx
http://www.sei.cmu.edu/library/assets/whitepapers/Cloudcomputingbasics.pdf
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.198.3684&rep=rep1&type=pdf
http://steinhardt.nyu.edu/scmsAdmin/uploads/002/926/StrumoloThesis.pdf
https://cloud.google.com/products/
Pitch Detection from Input Audio References
http://developer.android.com/reference/android/media/MediaRecorder.html
https://github.com/libgdx/libgdx/blob/master/extensions/gdx-audio/src/com/badlogic/gdx/audio/analysis/KissFFT.java
https://sites.google.com/site/piotrwendykier/software/jtransforms
http://epubl.ltu.se/1402-1617/2009/073/LTU-EX-09073-SE.pdf
http://www.fftw.org/
http://developer.android.com/reference/android/media/AudioRecord.html
http://andrewbrobinson.com/2011/11/27/capturing-raw-audio-data-in-android/
GUI Development
http://developer.android.com/guide/topics/ui/index.html
http://phandroid.com/2011/05/11/10-tips-for-android-ui-design/