ARC agent connect on linux servers with private link and proxy

Question

I'm trying to connect a linux server, in this case ubuntu 20.04 but we have other editions that have the same issue. I have been able to install azcmagent and have configured the proxy and the bypass for ARC. We have DNS in place that will resolve gbl.his.arc.azure.com and ae.his.arc.azure.com

When I run the connect command I get as far as

ERROR Failed to send config info to agent Error="read unix @->/var/opt/azcmagent/socks/himds.sock: i/o timeout"
ERROR Created Arc resource but couldn't connect it. You may delete the ARM resource and run azcmagent connect again to onboard. Error="read unix @->/var/opt/azcmagent/socks/himds.sock: i/o timeout"
DEBUG No proxy will be used: destination: gbl.his.arc.azure.com, bypass rule: arc
DEBUG Exiting...
DEBUG Starting client connection on: /var/opt/azcmagent/socks/himds.sock
DEBUG Awaiting status message from agent...
DEBUG Status Message received
DEBUG Cancelled change token
INFO Exit Code: AZCM0061: Agent Communication Error
INFO For troubleshooting, see https://aka.ms/arc/azcmerror
FATAL read unix @->/var/opt/azcmagent/socks/himds.sock: i/o timeout
DEBUG Exiting...

While watching the logs I observe the following

Get \"https://gbl.his.arc.azure.com/discovery?location=australiaeast&api-version=1.1-preview\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

When checking our firewall and proxy server I can see that gbl.his.arc.azure.com is trying to go through the proxy server

I have tried adding no_proxy="gbl.his.arc.azure.com,ae.his.arc.azure.com,global.in.ai.monitor.azure.com,api.monitor.azure.com,global.handler.control.monitor.azure.com,diagservices-query.monitor.azure.com,live.monitor.azure.com,profiler.monitor.azure.com,snapshot.monitor.azure.com"
but get the same error

I have restarted he himdsd service multiple times with the same error.

I have been able to get this working on at least 2 servers but I have a few that are still not working.

Answer 1

@Nick Murphy , Apologies for the delayed response. Based on the error, can you please check and ensure that proxy was set using azcmagent config set proxy.url cmd. The proxy details from environment variable (although could be inherited by the azcmagent cli tool when run from terminal), might not get inherited for the server process (himds etc.). Therefore, it is required to be set using the azcmagent config set proxy.url command. Please see this link for more details - Update or remove proxy settings

Also, for bypassing proxy for specific set of URLs, you may use azcmagent config set proxy. Bypass command. Please see the following link for details and usage example - Proxy bypass for private endpoints

Please let me know if you have any questions.