[SSH] Theoretically, if my personal computer is never compromised...

And neither is my SSH private key. SSH server is configured without password authentication.

Is it safe to say that there is virtually no chance my SSH server can be accessed without my authorisation?