In our Datacenter Operating System (DCOS), logs often contain vital information for understanding and debugging a cluster or the applications running on it. Today, we’re introducing three new commands for easy log access:
dcos node log
dcos service log
dcos task log
These are three main sources of logs in DCOS. We’re also introducing
dcos node ssh, which lets you SSH directly to a DCOS node without needing to look up IPs or hostnames.
The Mesos logs are systemd logs generated by the master and agent daemons. The master logs show resource offers and messages between the master and agents. The agent logs show resource utilization and container status (e.g..
docker pull status) on the agent node. Both show system errors such as resource exhaustion.
Previously, these were only available by SSH-ing into the machine and running
journalctl, or by going through the Mesos web UI. Now, you can simply run
dcos node log and print these logs directly on your local machine.
DCOS Services like Marathon and Cassandra by default print logs to
stderr. These logs contain anything the framework author chose to include, but often show task launches, resource accepts, and resource rejects. For example, if your Marathon app isn’t deploying, the answer might be in the Marathon logs.
dcos service log lets you directly access these logs, and lets you specify a custom log file if the service emits logs to a different location.
Finally, there are the task logs. Tasks are launched by DCOS services, and perform useful work. A task runs in a container on a single machine. Each Mesos task runs in its own “sandbox”, or a directory on the host machine. Files generated by this task appear in the sandbox, and Mesos itself creates
stderr files that capture these streams from the task.
Previously, these logs were only available by SSHing directly to the host, or by going through the Mesos web UI. You can now access them through
dcos task log. This command also supports logging multiple tasks at once. If you’re running a Marathon app that runs 10 instances of a web server for your hot new travel startup called “how-can-i-park,” you can simply run
dcos task log how-can-i-park, and you’ll see a single stream for all 10 log files.
We’re working toward surfacing much of the log information in a more structured way, and you can look forward to these monitoring tools in the future. For now, the logs are the sole source for a lot of important information, and new CLI commands make log access much easier.
Although Mesos emphasizes resources over individual machines, we know that a datacenter operator will often want to connect to a node to do some fine-grained debugging. So we’re also introducing
dcos node ssh, which enables anyone with SSH access to connect to a Mesos node.
dcos node ssh --master connects you to the leading master, and
dcos node ssh --slave=<slave-id> connects you to an agent.
Now there’s no need to go out of band to look up IP addresses or hostnames.
dcos node ssh connects to the private IP of the node, so you must run the command on the same network as your cluster. If you’re outside the network, consider setting up a VPN.
Visit our CLI documentation page to install or update to the latest version.