Software Protection Framework based on Code Obfuscation Techniques رسالة ماجستير

Abstract

Protecting intellectual property against tampering and reverse analysis is an urgent issue to many software designers, where illegal access to sensitive data is a form of copyright infringement. Software owners apply various protection techniques in order to address this issue. Many of used techniques are weak, since they are vulnerable to both dynamic and static analysis, where the other are very costly since they impose considerable performance penalties. Moreover, these techniques are often not good as they rely on “security through obscurity” which may deter some impatient adversaries, but against a dedicated adversary they offer little to no security. Thus, if an adversary succeeds in extracting and reusing a proprietary algorithm, the consequences will be significant. Moreover, reverse engineering remains a considerable threat to software developers and security experts. In this thesis, we proposed a software protection framework based on code obfuscation techniques in order to protect software against reverse analysis and unwanted modifications. First, we presented an obfuscation technique for java programs in order to protect software against static reverse analysis. The proposed technique integrates three levels of obfuscation; source code, data transformation, and bytecode transformation level. By combining these levels, we achieved a high level of code confusion, which makes the understanding or decompiling the obfuscated programs very complex or infeasible. Second, we proposed an obfuscating technique based on integrating encryption mechanism within recurrent neural network (RNN) in order to enhance the software protection level against dynamic analysis. Neural network provides a robust security characteristic in software protection, due to its ability of representing nonlinear algorithms with a powerful computational capability. The system is designed to enable the neural network generating of different encryptions for the same protected data. This creates a many to one relationship vi between the keys and the encryption. In order to complicate the reverse analysis of the software and hindering the Concolic testing attack, we train the neural network to simulate conditional behaviors of a program. Consequently, we replace the critical points of program’s data and control flow with a semantically equivalent neural network. Our method is designed to enable the neural network to execute conditional control transfers where the complexity of neural network ensures that the protected behavior is turned to a complicated and Incomprehensible form, making it impossible to extract its rules or locating the accurate inputs which lead to the execution paths behind the network. Third, we proposed a tamper resistance mechanism based on obfuscation and diversification. The proposed mechanism combined call graph obfuscating, stack obfuscating, diversification, memory layout obfuscating, randomization, and basic blocks reordering in order to thwart tampering and increase the difficulties of static reverse analysis and dynamic stack tracing analysis. A random mapping table is used for mapping the addresses of call and return instructions during the runtime of program. Moreover, a complex call graph of functions is generated to make the obfuscated program harder to attacker analyses and understanding due to a complex dependency of the obfuscated graph. Additionally, a hash mapping table are applied for encoding and decoding of the data stack frames during the runtime of program. The protection presented by our techniques is immune against static analysis, dynamic analysis, and tampering. Most tampering and revers analysis tools cannot easily undo the obfuscation effects of our techniques, as the attacker will consume a lot of time removing the bugs of the decompiled buggy program. Furthermore, our evaluations confirm that obfuscation effects in our system significantly increase the difficulties in revealing the obfuscated software. On the other hand, the performance evaluation confirms that our techniques protect software efficiently with an acceptable excess in execution time and memory usage.