Gfrktyl

My suggestion would be that you select a point on the asteroid to act as a pole. Perhaps the point of first landing? Then, using that point and asteroid's centre of gravity as references, you can map spherical coordinates.

Whoever is going to be on that asteroid will necessarily used radio communication to keep in contact with the rest of the crew.

To ensure communication a network of antennas has to be established, since a single antenna could at best serve half of the asteroid.

Each position can then be simply referred to the distance from the (closest) antennas.

I would suggest that they use a GeoHash which subdivides the asteroid into a hierarchical grid that can be navigated based on any desired granularity

https://en.wikipedia.org/wiki/Geohash

The origin point of the geohash should be the starting outpost location and this would provide a mostly sequential means of describing location where "most" objects that are physically close to each other, share similar geohash values.

NOTE: there are some minor cases where the hash of nearby locations will not be similar, but for most things it should be good enough.

Also NOTE: Geohash is a competing system of location to What3Words which is used here in Ireland and which produces non-sequential descriptors for location. This makes it impossible to know if two locations are close to each other just based on their 3 words which is why I would recommend using Geohash instead.

I'd maybe consider using the axis of rotation - it would be a very rare asteroid that isn't rotating somehow. Imagine sticking a skewer through the asteroid along the axis. That would give you a top and bottom, and then you can use spinwise and counter-spinwise (or something similar).

Obviously only works if you have an asteroid that is rotating nicely, something that is rotating a bit more chaotically might be more of an issue. If it's not rotating at all, or is spinning chaotically then Arkenstein XII's answer is definitely the way to go.

Depending on the length of your characters' stay on the asteroid (is it a mining operation, or are we setting up a habitat?), I'd say just establish a series of beacons that your personal navigation system can triangulate with. If it's longer-term, you have to set up magnetic shielding against the worst bits of solar rays anyway, so you might as well use that system and have a magnetic north as your standard.

An alternative to either of those would be to use the rotation of the asteroid as your north/south, and set up from there. If it's not spinning, this obviously wouldn't work, but it's worth bearing in mind. There are lots of ways to do it, but if this is a corporate thing, then they'd be likely to do whatever is easiest. That would be the triangulation beacon system, which they'd need for comms on a large, dense body anyway, and it would work regardless of why the asteroid is being navigated. It could be standard procedure.

You have to remember that even on Earth, apart from the poles and equator, the coordinate system is arbitrary. Greenwich isn't even a particularly important location in the grand scheme of things, apart from hosting the Royal Observatory that's used to define the 0 meridian.

The problem is the minimum number of points you need to define a coordinate system relative to a body. On a cube we conventionally select a vertex at one corner and three dimensions from that point. On a planet you take the axis of rotation to define the equator, and an arbitrary point on the surface to define a 0 meridian, everything is then defined relative to the equator, the 0 meridian, and the direction of rotation.

Asteroids are irregular. If it's tumbling then you could take the longest axis instead of the axis of rotation and the point of landing as 0 meridian, it could then be modelled as rotating around that axis, even if the axis itself is not stable. If it's spinning then you have poles and the landing point can be the 0 meridian. If you haven't landed on it yet then any arbitrary distinctive point can act as 0 meridian.

You can use the same coordinate system for celestial navigation (latitude and longitude) used by Terran mariners. What you need is : an almanac, a watch, a device for measuring the angle of the stars relative to some average horizon, and a map.

Almanac

The basic concept of celestial navigation is this : imagine several easily-recognizable stars. Next, imagine that you could draw a line from each of these stars that would pass through the center of whatever you are standing on. This line will touch the ground at one (only one) location. An almanac records these stars and the location (in latitude and longitude) of the point on the surface where the imaginary line from the star touches the ground.

Watch and Calendar

And, this spot will move as the object rotates around it's own axis (days); but will only move a little with the seasons.

Measurement Device (Sextant)

When you are standing on the spot where this imaginary line from your easily-recognizable star intersects the ground, that star will be directly overhead.

Map

Likely, you are not standing on one of these spots at any particular time. The angular measurement times the average radius of the asteroid provides you with the approximate circular (radial) distance between that point and where you are. Measure multiple stars to determine where these circles overlap on a map. That is your (approximate) location

Use Peaks and Valleys, and polar coordinates

In the old days, prior to GPS, cartography and mapping were major and necessary fields of study. Obviously without a magnetic pole you would be unable to create universal direction, but you could still create a 'map' by using surveying techniques found mainly in triangulation of mountain peaks.

Basically, you identify the highest point in a field of view, triangulate this with others to determine its distance and height, allowing you to build up an accurate map of peaks.

As it is an asteroid, it has a centre of mass, and should be relatively easy to determine which peak is the highest - this would be your reference point. The second tallest peak would be your reference base line, from which you could measure polar coordinates in a consistent anticlockwise direction from centre of gravity (z-axis) for all peaks thereon.

Often, when exploring Australia, Charles Sturt would have to 'artificially' create mini-piles of rocks to serve as references to determine coordinates. If your asteroid is flat, this could be used to create artificial reference points too.

Edit: To clarify the process:

As an example triangulation was used to map completely the Rocky Mountains in Colorado:

By using mountain peaks between 3 points, distances can be determined very accurately. Identify 2 peaks from one peak, measure the angle, then move to an alternate peak to measure the new angle between the first peaks. By plotting this on a map, you can use this to determine way finding, distance measures and reference points without necessarily relating to latitude / longitude. (It was also possible to use this technique without a compass).

Another example, giving each 'triangle' an identification in the Equadorian mountains:

By coding each triangulation, you can use this to describe locations. Pinpointing coordinates can be accomplished by mapping over a grid (polar or rectilinear) if necessary, perhaps based on the two tallest peaks as the origin.

This is a very flexible system, and was used in the 'old days' in particular when mapping difficult terrain, and is very accurate.

The asteroid would be mapped before anybody ever landed on it, rotation, procession etc would not, I think, be used due to the relative impact on these things from habitation and industry.

https://dawn.jpl.nasa.gov/multimedia/images/image-detail.html?id=PIA17480

It doesn't seem logical either to adopt magnetic polar attributes, as these are un-managed variables.

As with L Dutch & G.B. Robinson's answers, triangulation to communication antenna and 'relative stationary' satellites would be most useful, reliable & probably become ubiquitous. It's not like asteroid dwellers would be without electronic devices at any time, as their lives would depend on them. 3d mapping is not really any more complex, tho we would probably expect long term residents to develop abbreviations and colloquial terms as references

Maybe the easiest way would be to add a number of beacons and then just run bearings off, or between, them.

Possibly designate one, or a few, as Prime(s), analogous to magnetic poles, and relate the bearings of the others to them.

There are a number of navigation systems that use, or have used, fixed beacons to determine locations with varying degrees of accuracy depending on range from the beacons.

The center of mass would be the origin, all other coordinates (x, y, and z ~ latitude, longitude, elevation) would be pulled off that. You would orient and find the location of the CoM by triangulation of the stars as it rotates, with repeated measurements over time and in different places on the surface of the rock.

TL;DR: Two spikes, a ruler and a protractor

There is no perfect solution, just as there is no perfect solution for Earth. Witness differing data (as plural of datum, in which a datum is the basis of a specific system of reference). Here is a document describing conversion between two equally acceptable systems of datum: https://www.ngs.noaa.gov/CORS/Articles/WGS84NAD83.pdf

Here's a short snippet from that paper which goes to the heart of this difference:

First of all, one should understand that the 3-D Cartesian frames to
which the coordinates of the NAD83 and WGS84 refer are not identical.
Their origin, axes orientation in space, and the unit of scale
differ. Why? Simply, because the definitions of these two frames are
based on different sets of observations, processing algorithms, and
perhaps, geodetic assumptions.

So there is no one right answer for difficult surfaces -- we can't even get it settled rigorously for our relatively simple surface here on Earth. We simply agree which set of imperfections to try to work around. YET there must still be many more wrong answers than right answers.

Here's my proposed less-wrong answer, which is similar to many answers provided here, but perhaps more complete:

Select a north pole and drive a real or imaginary spike into it. Select an arbitrary prime meridien and drive another spike in where that intersects your chosen equator. We are unlikely to have a handy set of facts to support definitions like "equidistant at all longitude", so a naturally defined equator may not be available. Likewise, if the thing does not rotate appreciably, or if that rotation is perturbed or wholly inconsistent with an intuitional model of where poles and an equator "should" be, we can still get by with nothing more than our two chosen points -- a north pole, and a 0-0 point (corresponding to 0-0 in the Gulf of Guinea, check it out). With these two points we can always measure two facts:

• angle of rotation about the north pole from the prime meridien


• distance from the north pole as the space crow flies

These are not perfect, but can be agreed upon and figured by independent observers. Any radio distancing scheme could also be based upon these two points to yield the same two facts about position. This would also support map projection such as we are familiar with, including all of the existing shortcomings.

Let me start by clearing one thing - poles are not defined by magnetic field. Poles are defined by axis of rotation. Magnetic poles are somewhat independent from geographic poles and are only used as handy approximation, especially when you can't get a hold of more accurate measurement e.g of sun and stars, when the sky is clouded.
https://en.wikipedia.org/wiki/Poles_of_astronomical_bodies

As for your actual question - it depends on what you mean by large asteroid. Really big ones (bigger than 400km diameter) end up being nearly spherical due to their own gravity, we recently decided to call them dwarf planets. All dwarf planets rotate mostly regularly so you don't need to invent any new system, just choose a prime meridian, maybe a place of first landing, or the highest mountain, and you're good.
https://en.wikipedia.org/wiki/Dwarf_planet

Smaller, irregular asteroids, may not map nicely, especially if their rotation is very irregular, but you can still define the main axis by calculating mass distribution, choose the axis that gives maximum moment of intertia, which for regular bodies overlaps with axis of rotation, choose prime meridian (same as above), and you're good.
https://en.wikipedia.org/wiki/Moment_of_inertia

For very small asteroids, don't bother with Earth-like coordinates. If your total area is comparable with big cities like London, Moscow or Paris, do something similar. Define "districts" and "neighbourhoods" with memorable names. Place couple of beacons and plaques to make it clear which is where. Everybody will learn them after living there for couple of weeks.

In conclusion.

For any reasonably spherelike object you can define a
North Pole, where the major axis crosses the surface, choose an arbitrary Prime Meridian and plot an Earth-like grid of coordinates by simply projecting an imagined sphere on the actual surface. The asteroid doesn't even have to be very regular. These basic rules will work for all kind of potato shapes, as long as it has mostly positive surface curvature, which will always be the case except for very small asteroids. For those, you don't need a grid because the very small area and irregularity allows you to name all sectors unambiguously and just use those names.

You can apply a graph to your asteroid and find the distances of each point. From one another with this equation.

AB^2=(Bx-Ax)^2+(By-Ay)^2…………………^2=squared

If you make the same graph on four sided sections of asteroid you will have a rough grid.
Next

This is a better way to make a three dimensional coordinate site system labeling each point from the center as a distance with the equation:

The root of [(x2-x1)+(y2-y1)+(z2-z1)] where each point is a labeled coordinate at the surface of the asteroid.
Next

Use a circle grid to plot points from center with meters or kilometers or centimeters.

Then you find the angle degree of the point with this type of equation. Remember your circle grid always has a 90 degree angle already.

One can also find angle this way

Now apply the equation.
X^2 + y^2=(r cos @)^2. + (r sin @)^2 Where @ is a Greek representational letter for degrees or radians.

And this gives a unique coordinate for any point and it’s distamce from center of asteroid. You can put a stake or pin at any point and create a grid by stretching a bright colored string from one to the other. And use these shapes to find area.
Next

4. Another way is calculus if the asteroid is spinning you can find sections of surface area as zones of the asteroid.

And use integration.

For this you need the radius or diameter at each given integrated section. It if one can imagine a large amount of sections and each having a sub measure of length sections to each sliced section it also makes a grid.

There are few more possibilities and I will add if asked.

This is a rough idea and I am a little rusty and just woke up. Will edit later. I had a better presentation but the pictures would not copy here.

To elaborate on and combine a few other answers, consider that the Earth's magnetic north is not in a constant position, so it is not the factor in how lat/long coordinates are determined. Rather it is the axis of rotation as has been pointed out by others.

The equator is determined by being the great circle (circle which circumscribes the geoid of the planet), upon which twice per year (on the equinoxes) the sun travels exactly 180 degrees from horizon to horizon in a perpendicular fashion. Leap years are a thing because there is a slight difference in the time it takes for the earth to complete a single orbit of the sun vs. the period in which two equinoxes occur.

In your case a similar equator could be determined by observing the relationship between the body which the astroid orbits vs any 'wobble' along its axis of rotation (although this may take longer than a single 'year' or orbital period about the body.) The 2 points at which any two lines that are perpendicular to that equator intersect are its 'poles'. Lines which then intersect at a 90 degree angle at the poles can be used as meridians, the points at which those meridians intersect the equator are effectively arbitrary (the prime meridian is Euro-centric due to the fact that European cartographers defined it, not for any special geometric reason.)

The last step is then determined by math which describes projecting the surface of the asteroid to the abstract sphere that these lines define, surveying specific landmarks (natural or artificially placed) to act as well-known points which can be surveyed against as a reference system for any other point on the surface, allowing you to triangulate coordinates. These landmarks need to be offset along either the x or y axis from one another.

On a personal note I am also a fan of the answer which mentions 'geohashing' for modern cartographical applications. I have worked on an abstract specification known as 'Discrete Global Grid Systems' with the Open Geospatial Consortium which is conceptually similar and I'd encourage you to look into it if this sort of thing interests you. Its basis divides the sphere defined by an equator and meridians into a grid of triangles which is then projected to the surface of the body. Ironically, this is shown in the illustration of the robot in the left sidebar of this site. However your question specifically mentions coordinates, which a hashed addressing system does not provide.

Yes, you can. Cover the asteroid with a triangular grid of points and measure their relative locations, making the coordinate system you want to locate yourself within. You need it fine enough that you can see the three nearest points wherever you are. Trivially, if you make it fine enough, just knowing which triangle you are in is accurate enough, but we assume the triangles are not that small.

We know points $A,B,C$ and distances $a,b,c$ from the map. We want to locate $D$ by measuring the angles $ADB, BDC, CDA$

Using the measured angle $ADB$ we know from the inscribed angle theorem that $D$ is on a circle centered at $E$ on the bisector of $AB$. If angle $ADB$ is greater than $frac pi 2$ the center of the circle is outside the triangle and located so the angle subtended by $AB$ is $2pi-2ADB$. If angle $ADB$ is less than $frac pi 2$ the center is inside the triangle and the angle subtended by $AB$ is $2ADB$. We can construct two of these circles based on different sides of triangle $ABC$, find the intersection, and that is point $D$. In the diagram, $E$ is the center of the circle that $D$ is on. We assume $ADB gt frac pi 2$. That will be true for at least two sides of the triangle.

Given the coordinates of $A,B$, we find $F$ as the midpoint of $AB$. Then $AEF=pi-ADB$ and $EF=frac 12ABtan (pi-ADB)$. $D$ is on the circle with center $E$ and radius $AE$. Do the same for another side, find the intersection, and you are done.

Expanding a bit on LDutch's answer and MttJocy's comment on it. Take a page from pre-GPS aerial and nautical navigation systems.

Aviation commonly uses (fading as GPS is taking over, but still present) the VOR system and ADFs to figure positions and navigate. Essentially the various beacons all broadcast at a different frequency, with a directional pulse that varies in phase with the base signal. By calculating the phase difference between the primary and secondary phases your ADF provides your bearing from the station. Getting a bearing from two different stations gives you a very precise location. There are also some things that can be done in terms of phasing and signal strength to give an approximate distance from a single station, allowing for a rough position fix off of a single signal, but I am not aware of it being done in practice as the two bearing option is much more reliable. The major drawback for this system as currently implemented, and your intended use is that it is line of site and relatively short range (~200 miles). If you are too far from a station, or a pesky mountain (or the horizon of your asteroid) happens to be between you and it, you are not going to be able to pick up the signals. Meaning this system is not commonly used for surface travel.

The LORAN system was developed in WW2 and used until fairly recently for nautical navigation. It again works by using a series of fixed position transmitters, but in this case they come in paired units. Each member of the pairing is separated by a known distance, and they pulse out synchronized signals. The receiver compares the time difference in receiving the two pulses and from that difference you can plot a line of your relative distance from the two. Grab readings from an alternate pair to plot your location. The major advantage this system has over the other is range (~1500 miles) and being much less sensitive to LoS issues, though it is also somewhat less accurate. With the sensitivity and precision of modern electronics, it is conceivable that someone could build a 3+ point LORAN system and pinpoint their location pretty accurately off of one reading, the challenge being keeping the signals synchronized across the additional broadcast stations.

Now the range and LOS issues are primarily a function of the frequencies that are used in broadcasting the signals. So you could theoretically implement a VOR system in the frequency bands used by LORAN and have better performance in that respect, but it will also change the timing and phase calculations for determining direction. These would be engineering problems and may impact how quickly and reliably you could find you position. (Way too long since I studies this stuff to remember that level of detail)

Once the beacon network is established your coordinates become a range and bearing to a convenient beacon. "10 klicks from beacon XYZ bearing 175".

Note, I would advise against trying to establish a GPS network around an asteroid. That system is dependent on known orbits & timing for the satellites, and I suspect it would be very difficult to maintain such with the irregular shape and weak gravity of an asteroid.

Assuming a sufficiently advanced society, what about artificial poles, magnetic or otherwise?

A map with landmarks

The asteroid would be mapped out and photographed to generate a 3d model with landmarks

Like a pirate's map, directions would be given from the nearest landmark.

Mobile network sensors.
The OP clearly states that there are inhabitants. I'm going to make a leap of faith assuming that all the areas of interest have at least some inhabitant, or have had some visitor.

Local linear embedding has been used to reconstruct global maps from pairwise local distances. All you need is to equip the locals with peer-to-peer devices, measure the ping time, and resolve the local linear embedding problem, et voila: the map is ready. Bonus points, as long as the inhabitants don't move simultaneously in the same direction or according to some linear transformation, you can recompute their position over time and map unexplored areas of the asteroid.

Note that, as long as you can track changes in the pairwise distances between inhabitants, then you can define an "absolute" map, i.e. One where the thing you called North remains in the same place relative to the asteroid.

Any new inhabitant's location can be easily located on the map again by local linear embedding, or by trilateration from existing inhabitants.

References: http://www.cs.toronto.edu/~fritz/absps/globalcoord.pdf
http://www.pnas.org/content/early/2008/07/07/0709842104.abstract

Etc...