From the console you can disable X at startup by issuing the following command.
sudo mv /etc/init/lightdm.conf /etc/init/lightdm.conf.disabled
And then reboot the board.
You will then find you no longer have the X graphical system running and an extra ~100MB of memory to use.
If you wish to re-enable X at startup run the following to "undo" it.
sudo mv /etc/init/lightdm.conf.disabled /etc/init/lightdm.conf
And again following a reboot the service will be re-enabled.