google-app-engine


Error Reporting with App Engine Flexible Environment


I'm having troubles getting the Google Stackdriver Error Reporting to work using an App Engine Flexible Environment w/ Python 2.7
The documentation says a flexible environment requires manual configuration: https://cloud.google.com/error-reporting/docs/setting-up-on-app-engine
By default with the python runtime, it looks like google-fluentd is installed because ps ax | grep fluentd returns what the docs suggest. However, doing a sudo service google-fluentd restart fails.
I have fluent-logger==0.4.1 in my requirements.txt file.
I switched to a custom runtime to be able to place the forward.conf file in /etc/google-fluentd/config.d where the docs suggest.
My Docker file looks like this:
FROM gcr.io/google_appengine/python
RUN virtualenv /env -p python2.7
# stackdriver logging (for error reporting)
RUN mkdir -p /etc/google-fluentd/config.d
ADD forward.conf /etc/google-fluentd/config.d/
# Set virtualenv environment variables. This is equivalent to running
# source /env/bin/activate
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
ADD requirements.txt /app/
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
ADD . /app/
RUN python manage.py collectstatic --noinput
RUN python manage.py migrate --noinput
CMD gunicorn -b :$PORT project_name.wsgi
and forward.conf looks like:
<source>
type forward
port 24224
</source>
In my app I have a view that should report the error, but nothing comes through:
from django.http import HttpResponse
from django.views.generic import View
from fluent import sender
from fluent import event
import traceback
sender.setup('myapp', host='localhost', port=24224)
def report(ex):
data = {}
data['message'] = '{0}'.format(ex)
data['serviceContext'] = {'service' : 'myapp'}
# ... add more metadata
event.Event('errors', data)
class ErrorView(View):
def get(self, request, *args, **kwargs):
# report exception data using:
try:
Exception("Woops.. an Error Occurred")
except Exception as e:
report(traceback.format_exc())
raise e
Is there something I'm missing with the setup? The web server access logs come through just fine, but not exceptions or writing out to stderr or stdout
Update 5/9/2016
Thanks to #Steren's answer... it is much simpler than I made it out to be with Django in the Python Flexible Environment. There is no need for a custom runtime and installing fluentd. Below is a working sample that outputs a file log to /var/log/app_engine/custom_logs/error.log and a log formatter to put the log in the appropriate format.
LOGGING Settings:
'formatters': {
'gcp_json': {
'()': 'helloworld.gcp_logger.GCPJsonFormatter',
},
},
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/app_engine/custom_logs/errors.json',
'formatter': 'gcp_json',
},
},
'loggers': {
'django.request': {
'filters': ['require_debug_false'],
'handlers': ['file'],
'level': 'ERROR',
'propagate': True,
},
},
and the formatter:
import logging
import json
import os
class GCPJsonFormatter(logging.Formatter):
def format(self, record):
# https://cloud.google.com/error-reporting/docs/formatting-error-messages
return json.dumps({
'eventTime': record.created,
'message': self.formatException(record.exc_info),
'level': record.levelname,
'serviceContext': {
'service': os.environ.get('GAE_MODULE_NAME', ''),
'version': os.environ.get('GAE_MODULE_VERSION', ''),
},
"context": {
"httpRequest": self._get_request_info(record),
'user': str(record.request.user) if record.request else "",
"reportLocation": {
"filePath": record.pathname,
"lineNumber": record.lineno,
"functionName": record.funcName,
},
}
})
def _get_request_info(self, record):
try:
request = record.request
return {
"method": request.method,
"url": request.get_full_path(),
"userAgent": request.META.get("HTTP_USER_AGENT", ""),
"referrer": request.META.get("HTTP_REFERER", ""),
"responseStatusCode": record.status_code,
"remoteIp": request.META.get("REMOTE_ADDR", "")
}
except Exception:
return {}
There is a way to configure the App Engine flexible runtimes without requiring a custom runtime, it will soon be documented on the official documentation, and we are also working to make these steps simpler:
You should use similar code as the one detailed in the Google Compute Engine samples to send exception data as structured logs into a file named /var/log/app_engine/custom_logs/errors.json (do not send them via a TCP port)
It is important that:
The file path starts with /var/log/app_engine/custom_logs/
The file extension is .json
The file name contains err in the name.
The file only contains structured JSON objects.
You can then confirm in Stackdriver Logging that you see the errors appearing as structPayload in a custom log stream:
Pick App Engine in the first dropdown, and then custom.var.lop.app_engine.app.custom_logs.errors.json.
If this is the case, then Error Reporting should automatically starts processing these errors.
Let me know if it is not the case.

Related Links

How to build Many-to-Many relations in Google App-engine using JPA
Currently experiencing frequent DeadlineExceededExceptions when it was fine a few hours ago
How to get related to db.Model entity from datastore - what API allows it?
GAE-Python: Can Appstats record RPC made in background thread?
Blank Page after granting OAuth2 permission in BigQuery GAE Sample
GAE get Data using JDO with key
Setup GAE project with Resteasy using Maven in Eclipse
Working Maven3 configuration for AppEngine + DataNucleus + JPA 2
Spring Security ACL on App Engine Datastore
JDO 1:N issue (retriving data)
Endpoints Bad Api Configuration
Resolving an invalid memory address or nil pointer dereference
Design suggestion for an app which will wake up periodically and execute some long running tasks
Properly using map[string]interface{}?
Multiple calls to put() method when updating ndb on google app engine
GAE Can't set up a Resident Instance, related to new Migrate Settings option?

Categories

HOME
localization
paraview
delphi-7
ng-idle
nvd3.js
character-encoding
toolbox
solver
bitbucket-api
javacc
google-cloud-functions
iis-6
odoo-8
google-schemas
drop-down-menu
reactive-programming
stellar.js
qsqlquery
package.json
google-picker
xmonad
cfml
backpack-for-laravel
gulp-watch
exit
mount
fat
stackexchange
ms-access-web-app
google-cast
deltaspike
angularjs-material
sharepoint-workflow
latitude-longitude
modernizr
urlsession
numberpicker
skylink
branching-and-merging
congestion-control
http-request
pom.xml
robomongo
pdf-conversion
testrail
simplepie
xcode-server
windows-azure-queues
nikeplus-api
dbamp
phpwebsocket
exim4
libvlc
maybe
xenomai
pypiserver
oracle-spatial
django-urls
phonertc
amiga
apache-commons-daemon
phpldapadmin
java-security
mass-assignment
webmatrix-3
packet-sniffers
uncaught-typeerror
post-commit-hook
angular-carousel
wxformbuilder
gnumeric
terminate
fpdi
qscrollarea
hidapi
server-administration
android-sdk-2.3
mkv
patricia-trie
os.system
iboutlet
jdb
ikimagebrowserview
drawtobitmap
ajax-polling
interprocess
data-mapping
unmanagedresources

Resources

Encrypt Message