/
Disable Database Logging

Disable Database Logging

Owned by Justin, created
Last updated: Dec 05, 2023
5 min read

By Default, the Kasm deployment is configured to forward application logs into the PostgreSQL database. These logs power features within the Kasm UI such as the admin Dashboard reports, and the Logs widget which allows for viewing and searching the logs within the app.

These are handy features, but are only performant on smaller scale deployments. Depending on the activity and size of the Kasm deployment, the postgres database and the kasm_manager services may become sluggish and timeout during otherwise normal operations.

If the database server is sluggish or multiple timeout / queue full errors are observed (See error section below) , it may be time to disable database logging within the Kasm deployment. Administrators may desire to reduce the database log retention before fully disabling it. Instructions provided below

Logs can instead be forwandered to a SIEM of choice by ingesting the enriched JSON-based file logs or forwarding logs via HEC or HTTPS. See Kasm Native Logging for more details on these options.

Instructions

Disable Logging in the Global Settings

  1. Login to the Kasm UI as an administrator

  2. From the Admin panel open the Global Settings , by selecting Settings ->Global

  3. Update the following settings to 0 (zero):

    1. Debug Log Retention : 0

    2. Log Retention: 0

    3. Note: These settings represent the number of days to keep standard logs and debug logs. If the desire is to reduce the logging, set these values accordingly and monitor performance.

  4. For the changes to take effect you must restart the services on all WebApp servers.

    1. Open a terminal to the server and restart the services

    2. sudo /opt/kasm/bin/stop sudo /opt/kasm/bin/start

  5. After a few minutes the services should clear the logs from the database. You can verify they are empty by running a the following command from the database server. 0 indicates all logs have been cleared.

    1. sudo docker exec -it kasm_db psql -U kasmapp -d kasm -c "select count (*) from logs;" # Example Output count ------- 0 (1 row)

 

Disable Log Forwarding on Agents

By default, the Agents are configured to forward their logs to a manager, which will intern forward them into the database. Now that database logging is disabled, forwarding the logs is no longer necessary. If HEC or HTTP log forwarding is in use, this feature can remain on.

  1. Open a terminal on the Agent server(s)

  2. Stop the Kasm services

    1. sudo /opt/kasm/bin/stop

  3. Modify /opt/kasm/current/conf/app/agent.app.config.yaml

    1. sudo nano /opt/kasm/current/conf/app/agent.app.config.yaml

  4. Set forward_logs_to_manager: false and save the file

    1. #Example ... agent: auto_generate_kasm_docker_networks: false default_host_key: 1234 docker_port_listen_addr: localhost docker_script_timeout: 180 forward_logs_to_manager: true ...

  5. Start the Kasm services

    1. sudo /opt/kasm/bin/start

 

Potentially Related Errors

The following errors may be observed in the logs and indicate a poorly performing databasei if seen repeatedly.

Arguments: () --- Logging error --- Traceback (most recent call last): File "logging/handlers.py", line 1401, in emit File "logging/handlers.py", line 1363, in enqueue File "queue.py", line 190, in put_nowait File "queue.py", line 136, in put queue.Full Call stack: File "threading.py", line 890, in _bootstrap File "threading.py", line 932, in _bootstrap_inner File "threading.py", line 870, in run File "concurrent/futures/thread.py", line 80, in _worker File "concurrent/futures/thread.py", line 57, in run File "manager_api_server.py", line 604, in do_guardian File "provider_manager.py", line 1663, in delete_expired_anonymous_users File "logging/__init__.py", line 1434, in debug File "logging/__init__.py", line 1589, in _log File "logging/__init__.py", line 1599, in handle File "logging/__init__.py", line 1661, in callHandlers File "logging/__init__.py", line 954, in handle File "logging/handlers.py", line 1403, in emit Message: 'Looking for expired anonymous user accounts older that (1.0) hours' Arguments: ()

 

Arguments: () 2023-05-29 00:53:51,305 [INFO] manager_api_server: Processing Heartbeat From Server: (6136654b-4dd5-491d-8fd5-6e3e0d89ec51) --- Logging error --- Traceback (most recent call last): File "logging/handlers.py", line 1401, in emit File "logging/handlers.py", line 1363, in enqueue File "queue.py", line 190, in put_nowait File "queue.py", line 136, in put queue.Full Call stack: File "manager_api_server.py", line 1028, in <module> File "manager_api_server.py", line 960, in start File "tornado/platform/asyncio.py", line 132, in start File "asyncio/base_events.py", line 570, in run_forever File "asyncio/base_events.py", line 1859, in _run_once File "asyncio/events.py", line 81, in _run File "tornado/platform/asyncio.py", line 122, in _handle_events File "tornado/stack_context.py", line 300, in null_wrapper File "tornado/netutil.py", line 262, in accept_handler File "tornado/tcpserver.py", line 294, in _handle_connection File "tornado/httpserver.py", line 194, in handle_stream File "tornado/http1connection.py", line 722, in start_serving File "tornado/gen.py", line 326, in wrapper File "tornado/http1connection.py", line 735, in _server_request_loop File "tornado/http1connection.py", line 152, in read_response File "tornado/gen.py", line 346, in wrapper File "tornado/gen.py", line 1080, in __init__ File "tornado/gen.py", line 1147, in run File "tornado/http1connection.py", line 238, in _read_message File "tornado/httpserver.py", line 319, in finish File "tornado/routing.py", line 256, in finish File "tornado/web.py", line 2195, in finish File "tornado/web.py", line 2227, in execute File "tornado/gen.py", line 326, in wrapper File "tornado/web.py", line 1590, in _execute File "manager_api_server.py", line 798, in post File "logging/__init__.py", line 1434, in debug File "logging/__init__.py", line 1589, in _log File "logging/__init__.py", line 1599, in handle File "logging/__init__.py", line 1661, in callHandlers File "logging/__init__.py", line 954, in handle File "logging/handlers.py", line 1403, in emit Message: 'Updating existing server record: (6136654b-4dd5-491d-8fd5-6e3e0d89ec51)' Arguments: ()

 

Arguments: () --- Logging error --- Traceback (most recent call last): File "logging/handlers.py", line 1401, in emit File "logging/handlers.py", line 1363, in enqueue File "queue.py", line 190, in put_nowait File "queue.py", line 136, in put queue.Full Call stack: File "manager_api_server.py", line 1028, in <module> File "manager_api_server.py", line 960, in start File "tornado/platform/asyncio.py", line 132, in start File "asyncio/base_events.py", line 570, in run_forever File "asyncio/base_events.py", line 1859, in _run_once File "asyncio/events.py", line 81, in _run File "tornado/platform/asyncio.py", line 122, in _handle_events File "tornado/stack_context.py", line 300, in null_wrapper File "tornado/netutil.py", line 262, in accept_handler File "tornado/tcpserver.py", line 294, in _handle_connection File "tornado/httpserver.py", line 194, in handle_stream File "tornado/http1connection.py", line 722, in start_serving File "tornado/gen.py", line 326, in wrapper File "tornado/http1connection.py", line 735, in _server_request_loop File "tornado/http1connection.py", line 152, in read_response File "tornado/gen.py", line 346, in wrapper File "tornado/gen.py", line 1080, in __init__ File "tornado/gen.py", line 1147, in run File "tornado/http1connection.py", line 238, in _read_message File "tornado/httpserver.py", line 319, in finish File "tornado/routing.py", line 256, in finish File "tornado/web.py", line 2195, in finish File "tornado/web.py", line 2227, in execute File "tornado/gen.py", line 326, in wrapper File "tornado/web.py", line 1590, in _execute File "manager_api_server.py", line 790, in post File "logging/__init__.py", line 1446, in info File "logging/__init__.py", line 1589, in _log File "logging/__init__.py", line 1599, in handle File "logging/__init__.py", line 1661, in callHandlers File "logging/__init__.py", line 954, in handle File "logging/handlers.py", line 1403, in emit Message: 'Processing Heartbeat From Server: (6136654b-4dd5-491d-8fd5-6e3e0d89ec51)'

 

Arguments: () 2023-05-29 00:53:51,059 [INFO] tornado.access: 200 POST /api/v1/log (10.0.2.58) 25.62ms --- Logging error --- Traceback (most recent call last): File "logging/handlers.py", line 1401, in emit File "logging/handlers.py", line 1363, in enqueue File "queue.py", line 190, in put_nowait File "queue.py", line 136, in put queue.Full Call stack: File "manager_api_server.py", line 1028, in <module> File "manager_api_server.py", line 960, in start File "tornado/platform/asyncio.py", line 132, in start File "asyncio/base_events.py", line 570, in run_forever File "asyncio/base_events.py", line 1859, in _run_once File "asyncio/events.py", line 81, in _run File "tornado/platform/asyncio.py", line 122, in _handle_events File "tornado/stack_context.py", line 300, in null_wrapper File "tornado/netutil.py", line 262, in accept_handler File "tornado/tcpserver.py", line 294, in _handle_connection File "tornado/httpserver.py", line 194, in handle_stream File "tornado/http1connection.py", line 722, in start_serving File "tornado/gen.py", line 326, in wrapper File "tornado/http1connection.py", line 735, in _server_request_loop File "tornado/http1connection.py", line 152, in read_response File "tornado/gen.py", line 346, in wrapper File "tornado/gen.py", line 1080, in __init__ File "tornado/gen.py", line 1147, in run File "tornado/http1connection.py", line 238, in _read_message File "tornado/httpserver.py", line 319, in finish File "tornado/routing.py", line 256, in finish File "tornado/web.py", line 2195, in finish File "tornado/web.py", line 2227, in execute File "tornado/gen.py", line 326, in wrapper File "tornado/web.py", line 1569, in _execute File "manager_api_server.py", line 692, in prepare File "logging/__init__.py", line 1434, in debug File "logging/__init__.py", line 1589, in _log File "logging/__init__.py", line 1599, in handle File "logging/__init__.py", line 1661, in callHandlers File "logging/__init__.py", line 954, in handle File "logging/handlers.py", line 1403, in emit Message: 'Request authenticated (/api/v1/heartbeat)'

Related resources

Related content