To answer your main question from a “programmers perspective”:
As Keeper said, the way you seems like to proceed for encrypting your values consume a lot of memory and that’s normal.
Why ?
Since you’re fully mapping your data, key and output in memory first, you’re approximatively taking more than 3 times the data size in the heap (assuming all input/output are of the same size).
What you could do instead, is using a stream and process your encryption by block.
Here is a quick implementation of a Cypher XOR function that’s take its information from files:
(I made it in C# but I hope it’s understandable for you)
```
public void Encrypt(string dataPath, string keyPath, string outputPath)
{
System.IO.FileStream fsData = new System.IO.FileStream(dataPath, System.IO.FileMode.Open, System.IO.FileAccess.Read);
System.IO.BinaryReader rData = new System.IO.BinaryReader(fsData);
System.IO.FileStream fsKey = new System.IO.FileStream(keyPath, System.IO.FileMode.Open, System.IO.FileAccess.Read);
System.IO.BinaryReader rKey = new System.IO.BinaryReader(fsKey);
System.IO.FileStream fsOutput = new System.IO.FileStream(outputPath, System.IO.FileMode.Create, System.IO.FileAccess.Write);
System.IO.BinaryWriter wOutput = new System.IO.BinaryWriter(fsOutput);
int bufferSize = 1024;
byte[] bufferData = rData.ReadBytes(bufferSize);
byte[] bufferKey = rKey.ReadBytes(bufferSize);
while (bufferData.Length > 0)
{
byte[] bufferOutput = new byte[bufferData.Length];
for(int i = 0; i < bufferData.Length; ++i)
bufferOutput[i] = (byte)(bufferData[i] ^ bufferKey[i]);
wOutput.Write(bufferOutput);
bufferData = rData.ReadBytes(bufferSize);
bufferKey = rKey.ReadBytes(bufferSize);
}
wOutput.Close();
fsOutput.Close();
rKey.Close();
fsKey.Close();
rData.Close();
fsData.Close();
}
```
Notes:
- This is assuming data and key are of the same size.
- You can see the buffer size set to 1024, it will process the cypher method by block of 1024 bytes.
- Since system call are quite consuming, you should try to reduce them as much as possible so avoid to read/write only one byte into the stream (by the way, some stream implementation are semi intelligent and won’t actually write directly for one byte).
- For performance, you can estimate a buffer size depending the file size so you can map a small file directly into memory and process the big ones by blocks.
Cons:
- This method doesn’t allow you to process encryption that’s need the whole data content while processing, you need make sure it is feasible by blocks.
- You’re locking the files on the system during the time you’re using them so don’t forgot to close all handles at the end !
