Protecting the confidentiality of cryptographic keys in the event of partial or full system compromise is crucial for containing the impact of attacks. The Heartbleed vulnerability of April 2014, which allowed the remote leakage of secret keys from HTTPS web servers, is an indicative example. In this paper we present PixelVault, a system for keeping cryptographic keys and carrying out cryptographic operations exclusively on the GPU, which allows it to protect secret keys from leakage even in the event of full system compromise. This is possible by exposing secret keys only in GPU registers, keeping PixelVault's critical code in the GPU instruction cache, and preventing any access to both of them from the host. Due to the non-preemptive execution mode of the GPU, an adversary that has full control of the host cannot tamper with PixelVault's GPU code, but only terminate it, in which case all sensitive data is lost. We have implemented a PixelVault-enabled version of the OpenSSL library that allows the protection of existing applications with minimal modifications. Based on the results of our evaluation, PixelVault not only provides secure key storage using commodity hardware, but also significantly speeds up the processing throughput of cryptographic operations for server applications. Copyright is held by the owner/author(s).