Improve Payload CMS data fetching implementation

[m453h]

This pull request addresses an issue encountered when accessing content from the Payload CMS API deployed in production.

It was observed that requests with forwarded HTTP headers from the web application were being redirected to a 404 page, which pointed to the charter.africa site. This appears to be an unrelated issue that requires further investigation.
However, discarding (not forwarding) the HTTP request headers from the client (React Application) when making a request to Payload CMS via the Flask backend resolves this issue.

Minor improvements have been implemented by introducing a unified function to handle all Payload CMS API requests and error handling. Furthermore, the existing Flask view functions have been refactored to make use of this function.

[kilemensi]

👍🏽

It was observed that requests with forwarded HTTP headers from the web application were being redirected to a 404 page, which pointed to the charter.africa site. This appears to be an unrelated issue that requires further investigation.
However, discarding (not forwarding) the HTTP request headers from the client (React Application) when making a request to Payload CMS via the Flask backend resolves this issue.

Can we include an example dump of headers that cause redirect/errors?

[kilemensi]

Suggested change

	try:
	headers = {
	'Authorization': f'users API-Key {API_KEY}',
	'Content-Type': 'application/json'
	}
	if application_name:
	headers['Cs-App'] = application_name
	escaped_args = {k: html.escape(v) for k, v in (params or {}).items()}
	response = requests.get(url, params=escaped_args, headers=headers)
	response.raise_for_status()
	return response.json(), response.status_code
	except requests.RequestException as e:
	logger.error(f'Request failed: {str(e)}')
	return jsonify({'message': 'An internal error has occurred.'}), 500
	except ValueError as e:
	logger.error(f'Invalid JSON response: {str(e)}')
	return jsonify({'message': 'An internal error has occurred.'}), 500
	except Exception as e:
	logger.error(f'Unexpected error: {str(e)}')
	return jsonify({'message': 'An internal error has occurred.'}), 500
	headers = {
	'Authorization': f'users API-Key {API_KEY}',
	'Content-Type': 'application/json'
	}
	if application_name:
	headers['Cs-App'] = application_name
	try:
	escaped_args = {k: html.escape(v) for k, v in (params or {}).items()}
	response = requests.get(url, params=escaped_args, headers=headers)
	return response.json(), response.status_code
	except requests.RequestException:
	logger.exception('Request failed')
	except ValueError:
	logger.exception('Invalid JSON response')
	except Exception:
	logger.exception('Unexpected error')
	return jsonify({'message': 'An internal error has occurred.'}), 500

1. In a try/except, logger.exception includes e by default (unlike logger.error).
2. Do we need response.raise_for_status()?
3. Reduce empty lines

[m453h]

Cool, made the changes

Regarding 2. response.raise_for_status():
I think we we do need it, the motive was to throw a RequestException for the 404 response that is coming with HTML content (from charter.africa)...but I'm also open to let it throw an Invalid JSON response

[kilemensi]

I think we we do need it, the motive was to throw a RequestException for

Cool, cool... we can leave it there for better logs I guess.

[m453h]

👍🏽

It was observed that requests with forwarded HTTP headers from the web application were being redirected to a 404 page, which pointed to the charter.africa site. This appears to be an unrelated issue that requires further investigation.
However, discarding (not forwarding) the HTTP request headers from the client (React Application) when making a request to Payload CMS via the Flask backend resolves this issue.

Can we include an example dump of headers that cause redirect/errors?

Sure here are the dumps, I did more tests on this, it appears that the Host header is the one that is causing the issue

{
   "User-Agent":"PostmanRuntime/7.43.0",
   "Accept":"*/*",
   "Postman-Token":"*****",
   "Host":"localhost:8000",
   "Accept-Encoding":"gzip, deflate, br",
   "Connection":"keep-alive",
   "Cookie":"lng=en"
}

{
   "Host":"localhost:8000",
   "User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0",
   "Accept":"*/*",
   "Accept-Language":"en-US,en;q=0.5",
   "Accept-Encoding":"gzip, deflate, br, zstd",
   "Referer":"http://localhost:8000/",
   "Connection":"keep-alive",
   "Cookie":"lng=en;",
   "Sec-Fetch-Dest":"empty",
   "Sec-Fetch-Mode":"cors",
   "Sec-Fetch-Site":"same-origin",
   "Priority":"u=4"
}

[kilemensi]

Thanks @m453h. Based on the definition of host header, doesn't it sound like we need to change it? As in frontend will set host to Python APIs. The Python APIs need to "forward" this request to the actual final PayloadCMS & hence they must change host to the IP/domain of PayloadCMS.

... anyways, we can pick this up tomorrow.

[m453h]

Thanks @m453h. Based on the definition of host header, doesn't it sound like we need to change it? As in frontend will set host to Python APIs. The Python APIs need to "forward" this request to the actual final PayloadCMS & hence they must change host to the IP/domain of PayloadCMS.

... anyways, we can pick this up tomorrow.

Yes that's correct 💯, I made a slight change by just removing the header, and let Python requests set it, it does fix the issue

[kilemensi]

LGTM!