All you need to access the device, an Olive on VirtualBox in this case, is this config.
system {
services {
ssh;
telnet;
}
This config would allow ssh and telnet access to all interfaces.
Note: Root will not authenticate with telnet.
Now suppose you would like to block access on one or more interfaces.
You need a firewall filter.